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FOREWORD 


One  of  the  research  goals  of  the  Decision  Sciences  Laboratory  is  the 
development  of  design  principles  for  automated  training  subsystems 
which  could  be  built  into  future  Information  Systems.  Such  subsystems 
would  provide  Information  Systems  with  the  capability  of  training  auto¬ 
matically  their  own  operators.  To  be  able  to  design  such  a  capability 
requires  first  the  solution  of  many  conceptual  arid  experimental  prob¬ 
lems.  This  final  report  summarizes  design  principles,  equipment  and 
experiments  produced  under  Contract  AF  19(628) -455. 

This  report  is  one  in  a  series  supporting  Task  768204,  Automated 
Training  for  Information  Systems,  under  Project  7662,  Man -Computer 
Information  Processing.  The  research  was  conducted  from  1962  to  1964. 
The  Principal  Investigator  was  Dr.  Thomas  B.  Sheridan,  and  the  Con¬ 
tract  Monitor  was  Dr.  Sylvia  R.  Mayer. 

A  number  of  members  of  Bio -Dynamics'  staff  have  contributed  di¬ 
rectly  to  the  accomplishment  of  various  portions  of  this  study.  These 
include  T.  J.  Cummings,  B.  C.  Duggar,  A.  R.  Johnson,  J.  Mickunas, 

A.  W.  Mills,  and  R.  Rosenberg. 
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I.  INTRODUCTION 


The  training  of  operators  for  computer-based  systems  is  an  expensive 
undertaking.  Because  of  the  rapid  pace  of  system  evolution,  operator 
training  is  a  continuing  requirement.  Furthermore,  the  number  of  indi¬ 
viduals  receiving  an  identical  instruction  program  at  any  one  time  is  often 
extremely  small.  The  resultant  high  instruction/  student  ratio  require  - 
ments  escalate  training  costs  and  complicate  planning.  The  need  and  the 
potential  value  of  automated  training  for  improving  this  situation  has  been 
demonstrated  (1,  2,  3,  4,  5,  6). 

Air  Force  research  and  development  in  automated  training  has  pro¬ 
gressed  to  the  point  where  refinement  of  design  principles  and  specifica¬ 
tions  for  implementation  are  appropriate  research  objectives.  In  some 
instances,  "teaching  machines"  will  be,  or  are  being,  used  to  acquire 
skills,  and  simulators  are  being  used  to  exercise  these  acquired  skills. 
However,  present  and  future  Air  Force  Information  Systems  perform  func¬ 
tions  and  have  capabilities  which  overlap  the  requirements  of  automated 
instructional  systems  and  simulators.  Since  the  subject  matter  to  be 
taught  the  operators  of  information  systems  is  intimately  related  to  the 
specific  equipment  for  the  particular  task,  there  are  many  advantages  of 
involving  in  the  training  program  the  very  same  equipment  which  eventually 
will  be  used  on  the  job.  Because  of  this  overlap  of  operational  and  instruc¬ 
tional  requirements,  inclusion  within  the  operational  equipment  of  subsys  - 
terns  which  perform  this  instructional  function  appears  both  desirable  and 
feasible. 
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This  report  describes  the  results  and  conclusions  of  a  study  initiated 
in  July  1962  and  completed  in  December  1964  which  was  directed  at  the 
development  of  principles  for  the  design  of  automated  instruction  subsys¬ 
tems  for  Information  Systems.  A  series  of  four  Technical  Documentary 
Reports  have  been  issued  which  describe  in  detail  the  activities  and  results 
of  each  aspect  of  the  study  (7,  8,  9,  10).  This  report  brings  together  and 
summarizes  the  results  reported  in  the  individual  documents,  and  includes 
additional  items  which  did  not  warrant  separate  documentation. 

H.  APPROACH 

A  variety  of  tasks  were  selected  for  experimental  evaluation  of  instruc¬ 
tional  concepts  and  requirements.  The  three  tasks  used  in  the  major  exper¬ 
iments  were  characterized  by  three  widely  differing  skills: 

1.  Control  sequencing- -in  one  type  of  experiment,  the  operator  had  to 
perform  a  particular  sequence  of  actions  depending  upon  the  display  condi- 

I 

tions  (the  functions  of  the  controls  and  how  they  were  used  were  not  important). 

2.  Problem  analysis  and  selection  of  required  control  functions --in 
this  type  of  experiment,  the  subject  was  to  acquire  an  understanding  of  the 
functions  of  each  of  a  number  of  available  controls  and  how  they  interact. 

3.  Querying  and  reasoning- -in  this  experiment,  based  on  the  informa¬ 
tion  gained  from  answers  to  successive  queries,  the  operator  had  to  develop 
or  refine  hypotheses  and  select  additional  queries  with  which  to  test  them. 

Alternative  training  programs  were  developed  for  each  of  the  tasks 
and  experiments  designed  to  evaluate  them.  The  training  methods  were 
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selected  to  evaluate  several  dimensions  of  instructional  techniques: 

1.  Structuring  of  the  training  experience --ranging  from  operant  con- 

/ 

ditioning  to  formal  lecture /demonstration  presentations. 

2.  Program  order --the  "discovery"  principle  of  ordering  programs. 

3.  Display  techniques --taped  lectures  and  slides  versus  instruction 
manuals. 

4.  Language --words  versus  graphical  logic  flow  diagrams. 

It  was  then  possible  to  develop  some  general  design  principles  for  provid¬ 
ing  the  required  system  capabilities.  Since  computers  have  already  been 
shown  to  have  great  potential  for  controlling  instruction  programs  (11,  12), 
the  major  emphasis  of  this  study  was  on  exploitation  of  the  overlap  of  ex¬ 
ternal  features  of  operational  equipment  with  those  features  required  to 
teach  the  task. 

HI.  RESULTS 

A.  Experiments 

The  first  task  in  the  research  program  was  to  develop  a  graphi¬ 
cal-symbolic  description  of  an  exemplary  Air  Force  Information  System 
operator's  task.  The  SAGE  Intercept  Director's  job  was  chosen  for  this 
purpose,  partly  because  the  potential  value  of  automated  instructional 
techniques  had  already  been  demonstrated  in  this  context.  An  abstract 
hardware  model  was  subsequently  developed,  training  programs  devised, 
and  automated  instruction  carried  out  using  the  hardware  model  and  graph¬ 
ical  symbology.  It  was  determined  that  when  the  control  sequences  were 
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outlined  in  the  form  of  a  graphical  logic  flow  diagram  drawn  on  the  console, 
individuals  who  understood  the  symbology  required  no  task  instruction  (7). 
The  graphical  symbology  was  found  to  provide  a  concise  language  which  is 
an  efficient  medium  for  teaching  the  task  logic,  and  the  diagrams  were  cap¬ 
able  of  being  used  as  self -teaching  programs.  Additional  experiments 
demonstrated  that  the  task  could  be  taught  without  symbology  pre -instruc¬ 
tion  provided  that  response  evaluation  and  immediate  reinforcement  was 
given  to  the  trainee  (7).  However,  it  should  be  noted  that  console  layout 
must  correspond  topologically  to  the  task  logic  flow  diagrams  if  this  tech¬ 
nique  is  to  be  fully  exploited. 

In  a  later  experiment,  the  same  abstract  model  of  the  SAGE  IND  Di¬ 
rector's  job  was  used  to  demonstrate  that  a  taped  lecture  with  logic  flow 
diagrams  on  slides  which  were  projected  directly  onto  the  control  panel  of 
the  console  could  serve  as  a  performance  aid  or  teaching  program  (9). 

When  the  same  logic  diagrams  were  used  in  an  instruction  manual  to  sup¬ 
plement  written  text,  training  time  was  longer  than  with  the  tape  /slides 
program. 

A  series  of  experiments  were  conducted  using  a  querying/ reasoning 
task  to  evaluate  alternative  training  programs  and  performance  aids.  It 
was  found  that  both  practice  and  training  modified  performance.  When 
routine  bookkeeping  of  logical  deductions  was  automatically  provided  to 
the  naive  operator,  performance  was  as  good  or  better  than  after  any  of 
a  variety  of  training  methods  and  moderate  practice  (7).  It  was  also  de¬ 
termined  that  a  logic  flow  diagram  of  the  task  could  serve  as  an  efficient 
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and  sufficient  means  of  task  instruction  (9,  10). 

A  conceptual  approach  to  ordering  the  content  of  programed  instruc¬ 
tion  according  to  the  "discovery"  method  was  evaluated  using  a  special 
multi -control  console  and  display.  The  discovery  principle  was  used  to 
guide  the  trainee  through  a  phylogenic  evolution  of  system  capabilities  (7). 
Trainees  who  learned  using  the  discovery,  or  phylogenic,  organized  pro¬ 
gram  required  less  time  and  performed  better  than  did  those  who  used  a 
conventional  training  program  (9). 

B.  Logic  Flow  Diagrams 

Techniques  for  task  analysis  and  development  of  graphical  logic 
flow  diagrams  were  derived  (10).  The  potential  value  of  these  diagrams 
for  use  as  training  aids  was  demonstrated  for  several  widely  differing 
tasks  (7,  9,  10). 

C.  Engineering  Design  Principles  and  Considerations 

General  techniques  and  considerations  for  reducing  instructional 
requirements,  and  for  utilizing  portions  of  the  operational  equipment  for 
automated  instruction,  have  been  induced  from  the  experiments  and  from 
a  review  of  the  literature  (8). 

The  systems  capabilities  required  to  provide  the  training  functions 
used  in  the  experiments  vary  widely.  For  example,  at  one  extreme  the 
tape/ slides  program  could  be  presented  from  a  single  low-cost  tape  re¬ 
corder  and  projector  without  connections  of  any  sort  to  the  operational 
equipment  (8,  9).  However,  the  same  program  would  be  more  effective. 
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and  more  costly,  if  the  taped  messages  and  slides  were  selected  by  the 
computer  based  on  an  evaluation  of  each  successive  operator  response. 

In  the  latter  case,  access  to  the  computer  program  must  be  available, 
and  additional  inputs  provided  to  the  computer  from  the  individual  con¬ 
trols.  With  respect  to  the  use  of  a  discovery  or  phylo  genic  ally  ordered 
program,  the  required  system  capabilities  are  much  more  severe  (9). 
Means  for  explicitly  showing  the  need  for  each  control  function,  identify¬ 
ing  the  control  device,  and  for  demonstrating  the  function  can  be  provided 
from  a  program  external  to  the  operational  equipment  which  merely  dis¬ 
plays  situations  according  to  a  planned  sequence  on  command.  Require¬ 
ments  for  response  evaluation  have  not  yet  been  thoroughly  investigated. 
The  verbal  responses  used  in  the  experiments  could  not  be  reliably  eval¬ 
uated  by  present  state-of-the-art  equipment. 

The  economics  of  providing  self-instructional  features  are  highly 
dependent  on  the  number  of  operators  who  will  require  training  during  the 
effective  life  of  the  equipment.  For  conventional  training  programs,  the 
direct  costs  consist  of:  instructor's  salaries,  instruction  materials 
(manuals,  mock-ups,  simulators),  facilities  (classrooms,  fixtures,  light, 
heat,  etc.  ),  cost  of  time  spent  in  training,  and  costs  of  updating  the  in¬ 
structors  and  materials  for  future  systems.  With  self -instructional  sys¬ 
tems,  the  costs  are  in  equipment  modification,  program  preparation, 
cost  of  time  spent  in  training,  and  cost  of  modifying  the  equipment  and 
program  for  future  systems  (assuming  that  stand-by  systems  components 
are  used). 
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D.  Equipment  Constructed 


In  the  course  of  this  study,  three  items  of  equipment  were  con¬ 
structed.  The  first,  the  console  used  for  the  abstract  model  of  the  SAGE 
IND  Director's  task,  has  been  previously  described  in  detail  (7).  The 
second,  the  phylogenic  console  for  controlling  Lissajous  figures,  has  also 
been  described  in  an  earlier  report  (9).  The  third  item,  a  device  for  dem¬ 
onstrating  probabilistic  information  processing,  was  not  used  experimen¬ 
tally  and  is  described  in  the  Appendix  to  this  report. 

IV.  CONCLUSIONS  AND  RECOMMENDATIONS 

1.  The  graphical  symbology  developed  in  this  study  provided  a  con¬ 
cise  and  efficient  language  for  teaching  the  task  logic  for  several  represen¬ 
tative  tasks.  It  can  be  used  independently  or  with  other  material  in  instruc¬ 
tional  programs.  It  is  recommended  that  graphical  logic  flow  diagrams  be 
tried  as  teaching  aids  in  an  actual  Air  Force  training  situation. 

2.  Design,  layout,  and  labeling  of  consoles  in  a  topological  relation 
to  the  task  logic  flow  diagrams  will  facilitate  implementation  of  self -teach¬ 
ing  functions. 

3.  Performance  aids  and  teaching  aids  beneficially  modified  behavior 
during  a  querying-reasoning  task.  Provision  of  automated  bookkeeping 
greatly  reduced  training  requirements,  while  provision  of  a  task  descrip¬ 
tion  which  explicitly  showed  the  contingencies  governing  each  successive 
decision  produced  the  most  rapid  performance  improvement. 
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4.  A  phylogenic  presentation  of  control  function  ordered  according 
to  the  discovery  principle  is  an  effective  format  for  instruction.  Because 
of  the  potential  value  of  this  concept  and  present  implementation  problems, 
it  is  recommended  that  further  research  be  directed  towards  refinement 
and  ease  of  exploitation. 

5.  Slide/tape  instruction  programs  can  be  used  to  provide  on-the-job 
training  or  serve  as  performance  aids  with  operational  equipment.  Since 
this  technique  can  be  evaluated  in  the  field  without  requiring  equipment  mod 
ification,  it  is  recommended  that  such  a  program  be  developed  and  used  to 
meet  an  existing  training  need.  Layout  of  future  consoles  to  facilitate  use 
of  this  technique  should  be  deferred  until  the  field  evaluation  is  completed. 

6.  Varying  degrees  of  self-instructional  capabilities  can  now  be  in¬ 
cluded  in  Air  Force  Information  Systems.  Design  principles  and  techniques 
should  be  disseminated  and  human  engineering  requirements  modified  to 
reflect  the  necessity  for  giving  specific  consideration  to  inclusion  of  self- 
instructional  capabilities. 
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VI.  APPENDIX 
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An  Inexpensive  Probabilistic  Information  Processor  (PIP) 

As  more  sophisticated  military  information  systems  evolve,  the 
users  will  gain  access  to  a  variety  of  probabilistic  data.  These  data  will 
facilitate  decision  making  if  properly  interpreted.  The  mathematics  of 
probabilistic  information  processing  according  to  Bayes  rule  are  well 
known  and  computers  have  been  programed  for  this  purpose.  For  situa¬ 
tions  in  which  computers  are  not  available,  the  mathematical  solutions 
can  be  computed  manually. 

If  future  information  systems  and  sensors  are  to  be  fully  exploited, 
users  at  all  levels  should  be  familiar  with  the  fundamentals  of  Bayes  rule. 
Simple  inexpensive  devices  for  computing  probability  data  are  needed  to 
serve  as  training  aids,  or  as  performance  aids  in  situations  where  access 
to  full-size  computers  is  not  available. 

An  inexpensive  and  simple  device  for  computing  probabilities  accord¬ 
ing  to  Bayes  rule  is  described  in  the  following  paragraphs.  Cost  of  com¬ 
ponents  is  kept  low  by  using  potentiometers  for  multipliers  and  the  human 
operator  as  a  nulling  device. 

For  computing  the  successive  probabilities  of  alternative  hypotheses, 

given  successive  items  of  data,  Bayes  rule  takes  the  form: 

p  (D.  |  H.)  p  (H.) 

£  P  (Dj  |  H.)  p  (H.) 


where:  H.  is  the  inclusive  set  of  all  mutually  exclusive  "hypotheses" 
(causes). 

D.  is  the  set  of  data,  observations,  or  "effects",  where  any 

^  of  the  D.  can  be  the  effect  of  several  H.. 

J  i 

p  is  the  probability. 

Given  all  present  and  past  evidence,  this  equation  is  of  the  form: 

a.*  b. 

Probability  of  a  specific  hypothesis  i  =  ,  the  solution  of 

I  ai*  i 

which  serves  as  the  a  priori  probability  for  the  next  step  (i.  e.  ,  the  new 


p  (H.)). 

Two  potentiometers,  each  with  two  gangs,  are  arranged  such  that  the 
voltage  out  of  one  gang  of  potentiometer  1  is  p  (FT)  =  b^  (there  are  as  many 
sets  of  potentiometers  as  there  are  H.).  The  output  of  the  first  gang,  b^, 
is  then  multiplied  by  a^  with  a  second  potentiometer  (this  is  a  manual  set¬ 
ting  of  p  (D^  |  FT)  using  a  calibrated  dial).  All  of  the  a.b.  are  then  added  in 
an  analog  adder  (requires  one  operational  amplifier).  Thus,  both  the  num¬ 
erator  and  denominator  of  the  function  p  (FT  |  D.)  have  been  computed.  The 
£a.b.  is  then  applied  to  ganged  potentiometer  number  2,  and  the  wiper 
adjusted  manually  until  its  voltage  (in  terms  of  some  current  null  instru¬ 
ment  or  ammeter)  equals  a.b..  When  this  adjustment  is  made,  the  other 

Si*  ^i 

gang  of  this  potentiometer  shows  a  voltage  s? — g-  on  its  wiper  by  simple 

a  i  i 

proportions. 
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The  equipment  can  be  packaged  in  an  arrangement  such  as  shown 
below. 


/  ° 

/7 

7^ 

z 

0 

© 

0 

ZP(H^ 

© 

pC°j) 

p(H,) 

D 

a  •  o 
b 

p(H2) 

KryA 

b~* — | 

Knob  for  setting  p  (D  |  H.) 
Null  device  for  multiplier 
-Knob  for  nulling 
Switch 


The  box  marked  "0"  on  top  would  contain  two  analog  adders,  one  to 

sum  up  the  p  (H^)  at  each  step  to  confirm  to  the  operator  that  the  sum  was 

equal  to  unity,  and  the  other  to  sum  over -all  p  (D.  H.)  p  (H.).  Boxes 

J  1  ^ 

marked  1,  2, - would  be  identical.  Each  would  display  the  present  prob¬ 

ability  of  any  FL,  based  on  all  previously  processed  Dj.  The  top  row  of 
the  diamond  of  knobs  is  used  by  the  subject  to  indicate  his  estimate  of 
p  (H-  |  D.).  The  other  three  knobs  are  for  a  perfunctory  operation  which 
the  operator  performs  after  he  has  gone  down  the  line  and  set  all  of  the 
p  (H;  I  D.)  knobs  for  that  round. 

Operation  proceeds  as  follows: 

1.  Set  a  priori  p  (H^). 

2.  Adjust  potentiometers  for  p  (D.  |  H.)  on  each  box. 

3.  Connect  the  output  of  the  adder  to  pot  2  and  adjust  the  wiper 
to  null  the  product  of  steps  1  and  2  (do  this  for  each  H.  box 
down  the  line). 


4.  Switch  over  to  the  right-hand  side  of  pot  1  and  the  left-hand 
side  of  pot  2  and  repeat  the  cycle  of  operations  with  new  data 
inputs  (the  voltage  on  the  left  gang  of  pot  2  is  the  new  b.,  and 
the  multipliers,  analog  adders,  and  null  instruments  are  now 
connected  between  this  gang  and  the  right  gang  of  pot  1). 
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A  circuit  diagram  for  a  single  H.,  unit  and  box  is  shown  below. 


Hi  UNITS 


H0  UNIT 
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Rj  and  ganged  and  low  impedance. 

Rg  and  R^  ganged  and  low  impedance. 

R  (for  setting  p  (D.  H.)  is  medium  impedance. 

5 

All  R's  in  the  adders  are  equal  and  of  high  impedance. 

Switches  S1 ,  S  ,  and  S  are  ganged. 

A  breadboard  model  of  a  two  hypothesis  PIP  was  constructed  for  dem¬ 
onstration  purposes.  It  was  found  that  inaccuracies  can  arise  when 
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inexpensive  linear  pots  must  be  used  at  0-5%  or  95-100%  of  their  rotation 

ranges.  However,  in  the  middle  ranges,  the  system  approached  slide 

rule  accuracy.  For  other  than  demonstration  or  teaching  purposes, 

ganged  precision  potentiometers  would  be  required.  The  ratios  of  R_  to 

R.  and  the  adder  R's  to  Rr  should  be  in  the  order  of  50  to  1,  and  the  meter 
1  5 

resistance  at  least  as  large  as  the  adder  R' s  to  achieve  a  moderate  degree 
of  precision. 
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